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Abstract 



The row model for frustrated XY spins on a triangular lattice in 2D 
is used to study incommensurate (IC) spiral and commensurate (C) an- 
tiferromagnetic (AF) phases, in the regime where a (C)-(IC) transition 
occurs. Using fluctuating boundary conditions and specific histogram 
techniques, a detailed Monte Carlo (MC) study reveals more structure 
in the phase diagram than found in previous MC simulations of the full 
parameter space. On the (C) side, equilibrium configurations consist 
of alternating stripes of spiral phases of opposite chirality separated by 
walls of the (C) phase. For this same parameter regime, thermody- 
namic quantities are computed analytically using the NSCHA, a gen- 
eralization of the self consistent harmonic approximation appropriate 
for chiral systems. On the commensurate side of the (C)-(IC) bound- 
ary, NSCHA predicts an instability of the (C) phase. This suggests 
that the state is spatially inhomogeneous, consistent with the present 
MC result: it resembles the smectic-A phase of liquid crystals, and its 
existence implies that the Lifshitz point is at T = for modulated XY 
spins in 2D. The connection between frustrated XY systems and the 
vortex state of strong type II superconductors suggests that the smec- 
tic phase may correspond to a vortex liquid phase of superconducting 
layers. 

PACS Number 75.10.Hk 
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I. INTRODUCTION 



Frustration is an ubiquitous phenomenon in condensed matter physics. It occurs 
whenever several ground states of a system compete at different length scales. Ex- 
amples of such a situation are non-interacting electrons in a tight binding potential 
subjected to a uniform magnetic field!, networks of superconducting wires! or of 
Josephson junctions! in a field and spins with competing interactions!. In particu- 
lar, frustrated magnetic systems have been used in the quantum realizations 
of the spin liquid state! (advocated in the context of high T c superconductors). In 
the classical case, XY (0(2)) spins model the vortex state of layered, strong type 
II superconductors!'!. Frustration manifests itself by the existence of chiral vari- 
ables. The effect of this additional (Z2) symmetry on phase transitions is still an 
open debate. For fully frustrated models there remains to establish whether the the 
Z 2 and the 0(2) symmetries are broken at different temperatures or at the same 
temperature!^. In the context of 2D helimagnets this issue comes up when one 
studies the commensurate-incommensurate ((C)-(IC)) transition!: on the (C) side, 
the state is non chiral whereas on the (IC) side chirality is coupled to the XY vari- 
ables. Insight into this particular problem can be gained by studying the phase 
diagram of the row model, an anisotropic frustrated 2D XY model on a triangular 
lattice: it is a generalization of the fully frustrated XY model on the triangular lat- 
tice (FFTXY) where all the bonds strengths J are multiplied by rj in the horizontal 
directionEl^El (the FFTXY model corresponds to 77 = 1). 

In order to study this systemS and other (IC) structures^, a MC algorithm with 
"self determined (Fluctuating) Boundary Conditions" (FBC) was developed. The 
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resulting rj versus T phase diagram indicated a continuous (C)-(IC) transition line 
starting at rj — 0.5 for T = and ending at a Lifshitz pointEl (LP) for t]l ~ 0.62 and 
Tl ~ 0.42J. Besides, increasing T at fixed 77 (0.5 < t] < r\£) produced the following 
sequence of phases: an (IC) state at low T; then, across the (C)-(IC) transition, 
at Tc-ic, one moves into the (C) phase; lastly one reaches the paramagnetic (P) 
boundary at Tp. In this process one of the eigenvalues of the spinwave stiffness 
matrix decreases uniformly as T varies from zero to Tc-ic, vanishes at Tq-ici 
increases again in the (C) phase and becomes zero above Tp. 

These MC results raise an issue, because they yield a thermodynamically stable 
commensurate state for T > Tc-ic and also because they predict a finite temper- 
ature LP: from the standpoint of critical phenomena, the (C) phase is in the same 
universality class as the ferromagnetic XY model; in the vicinity of the LP one may 
analytically compute the bare (unrenormalized) stiffness constant and one finds that 
it is very small. In this regime, the KT renormalization group equationstiiE show 
that vortex-antivortex pairs are unbound, implying that the (C) phase is thermody- 
namically unstable near the (C)-(IC) transition; so renormalization group predicts 
a reentrant (P) phase and thus a zero temperature Lifshitz point, at variance with 
the Monte Carlo results. Another indication that the LP may occur at T = comes 
from the study of 2D modulated O(N) spin systems exhibiting a (C)-(IC) transi- 
tion; renormalization group analysis predicts^ that the LP is at T = whenever 
iV > 2; numerical studies show that this also holds if N = 1 (ANNNI model)il. 
Interpolating to the case N = 2 one might have then expected a zero temperature 
LP for 2D XY systems: using a phase-only Hamiltonian, Garel and Doniach indeed 
reached this conclusion for the so-called J\ — J2 modeS 
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The present paper reconciles these a-priori conflicting results: 
In section [H], we present a MC algorithm allowing to study incommensurate and 
spatially inhomogeneous statesB it combines FBC and specific boundary condition 
histograms designed for FBC. This approach allows us to analyse the data near the 



(C)-(IC) transition. Section III presents a MC study of the row model for 
0.5 < 7] < t]l, for various T and rj near the (C)-(IC) transition. Special attention is 
devoted to the " (C) phase" for T > Tc-w- Our results suggest that the equilibrium 
structure is spatially inhomogeneous: Fig (|6]) shows a striped structure, correspond- 
ing to the coexistence of domains of opposite chirality separated by walls of the 
collinear phase. Such a state resembles the smectic-A phase of liquid crystals. In 
this regime, we find that j xx - the spin rigidity in the horizontal (77-bond) direction 
- is zero, whereas 7 yy - the spin rigidity in the vertical direction - is strictly positive 
(Fig (0)). Stripes exist because the coupling between phase and chiral variables is 
relevant at all T when rj > 0.5. This coupling helps explain why domains of the 
chiral phase are present for T > Tc-w- Moreover, fluctuations between a spatially 
homogenous state (the incommensurate phase) and a spatially inhomogeneous spi- 
ral domain state (the striped phase) do not allow simple scaling analysis of critical 
quantities at T C -ic (Fig (§)). 

These observations allow us to conclude that, in the phase diagram, the (C) and 
(IC) phases are separated by a smectic-like phase, and only come in contact at T = 
and 7] = 0.5, so that the LP is indeed at T = for the 2D XY model. On the other 
hand there is no re-entrant (P) phase between the (C) and (IC) regions. 

Our numerical findings are further supported by analytic calculations, presented 



in section |TV|. These use the NSCHA method (New Self-consistent Harmonic Ap- 
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proximation), a recently developed variational approach for frustrated sysfr 
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II. MONTE CARLO 



A. Fluctuating boundary conditions. 

For incommensurate phases, the choice of periodic boundary conditions (PBC) 
in a MC simulation is not suitable, since these break the magnetic symmetry of the 
system. 

Instead, self-consistent boundary conditions, using FBC, have been proposed 
to overcome the problemlilli!. The main feature of FBC is to add new dynamical 
variables A a (a = 1,2, ...,.D where D is the dimensionality of the lattice ) 
corresponding to a shift at the boundaries. In equilibrium the new "boundary 
variables" A a will fluctuate around their most probable value A°. For an L x L 
system of XY spins on a lattice, the FBC method amounts to imposing the following 
constraint on the phases 9(f) of the spins, at the boundary 

9(f + nLu x + mLuy) = 6(f) + nLA x + mLA y (1) 

Using FBC allows us to preserve translational invariance: performing a change of 
variables 

9(f) = <p(f) + A.f (2) 

the constraint on (p becomes 

ip(f+ nLu x + mLuy) = <p(f) (3) 

In terms of the new variable ip the partition function of the L x L system with FBC 
is: 
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Z FBC = L 2 d 2 A {j - d(pie~ P '^~^ iJ ' h] cos( ^"^~ A - (rl ^ )) ) (4) 

Zfbc can be factorized as a product of a set of partition functions, Z(A), each one 
corresponding to a fixed shift A at the boundaries: 



Z FBC = L 2 Z(A)d 2 A = L 2 d 2 Ae-? L2f ^ (5) 

J-n/L J-n/L 



where /(A) is the *r periodic free energy density associated with the shift A at the 
boundary: /(A) = -T ln(Z(A))/L 2 . 

— * 

For a system with a helical phase at low temperature, /(A) displays a minimum 
for A = A and for a spiral phase, the pitch Qq, is the ^ determination of A such 
that (p(r) — in equilibrium (see Eq.^|). Since the main contribution to the integral 
(Eq.|5|) comes from A = Aq, the components j xx ,7 ra of the spin rigidity^ are given 
by 



5 2 f(A) 



SAl 



yy ^/(A) 



SAl 

AO v 



(6) 

A° 



where p is a (lattice dependent) geometrical factor. 

At low T and far from the (C)-(IC) boundary (where j xx = 0), f3^ xx » 1 and 
P^yy » i u sing Eq | anc i Eq .| tnen g i ve sffl 

= - >y yy = - (7) 

7 Li XA J 7 L 2 xa v [) 

where xa, = P < (A x -A° x ) 2 > (resp. x/\ y = P < (A y -A° y ) 2 > ) is the susceptibility 
for A x (resp. A y ). 
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B. Boundary condition histograms: A— Histograms 

In the previous section we showed that the partition function with FBC is a 
sum over partition functions Z(A). A practical way to perform this sum is to count 
the number of configurations obtained for each of the allowed values of A x and A y . 
Since A x and A y are defined modulo ^ , this can be easily done by histograms in 
A x and A y , which we call A— histograms. 

Denoting by -P(A) = P(A X , A y ) the probability distribution for A, the A- 
histogram free energy density is obtained from: 

/(A) = l n (p(A)) + Constant (8) 

If /(A) has a deep minimum for A = A , the zeroes of the first derivative of the free 
energy yield the value of A . The second derivatives of the free energy computed for 
A = A give the components of the spinwave stiffness 7, by Eq.^j. But even if P(A) 
is not sharply peaked (see below), histograms allow to compute any thermodynamic 
observable as an average over P{A). 

This algorithm is especially useful when (i) one approaches a critical (C)-(IC) 
transition: A undergoes large fluctuations and Eq.[7| breaks down; histograms give 
much more accurate results and are well suited to scaling analysis (ii) equilibrium 
configurations correspond to inhomogeneous structures: in that case, histograms 
yield multi-peak structures. For instance, if domains of the (C) and (IC) phases 
coexist near Tc-ic the free energy will display minima at A = and at ±A°. 
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III. NUMERICAL ANALYSIS OF THE ROW MODEL NEAR THE 

(C)-(IC) TRANSITION 



Since the incommensurability is only present in the x (rj— bonds) direction we 
used hybrid boundary conditions: PBC in the y direction and FBC in the x direction. 
A standard Metropolis algorithm was applied to the spin angles and to the boundary 
shift in the x direction. Lattices sizes ranged from 18 2 to 48 2 and the number of 
MCS/spin was of order 10 5 — 10 6 . Typically the first 10 4 steps were discarded for 
equilibration. In contrast to our previous study of this systemEl, A— histograms were 
included here. These were used to determine Qo (the x component of the wavevector) 
as well as the spinwave stiffnesses along x and y. In addition, we monitored 

• the staggered chiralities S = (a) with 

a= 1 y E{ki)eP°ki (g) 
Npfaj: { ki}eP<7ki{T = 0) 1 ; 

where P refers to plaquettes in the same chiral state at T = 
and 

<Tki = ^-(.9k-9i) (10) 

Z7T 

(for Eq.[l0|, the angular determination of the term in parenthesis is taken in 
the interval [— tt, +tc] (see Ref.[l(])). 

• the chiral susceptibility 

x. = ^ 2 -£ 2 ) (ii) 
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the Binder order parameter for chiralities 



1 
2 



V)' 

> 2 >, 



;i2) 



A. Study of the (C)-(IC) line at fixed rj 

In the phase diagram of Fig ([]]), AL is a line separating the spiral incommen- 
surate phase from the commensurate layered antiferromagnetic (C) phase. It is 
characterized by a divergence of the chiral succeptibility and by the continuous van- 
ishing of the x component of the spin stiffness (Fig (Q)). The y component of the 
spin stiffness, on the other hand, does not show any non-analiticity near AL. In this 
part, we keep the value rj fixed and we vary the temperature. Typically we chose 
7] = 0.575 and i] = 0.55. Starting from the low temperature phase, we observe that 
^xx _^ q anc j ^at simultaneously the chiral susceptibility diverges as one approaches 
AL, Fig (U). This behavior can be understood as follows: Eikmans et al's Coulomb 
gas analysis of the generalized Villain modeS, when generalized to the row model, 
gives0: 

j xx oc — (13) 
X* 

Chiral variables and spin angle variables are coupled in the (IC) phase; thus ^ xx 
can go to zero in a continuous fashion, rather than jump, on crossing AL. Similarly 
from the same Coulomb gas analysis, one expects that 7™ is well behaved across 
AL (Fig (Q)). Fig ([|) shows that Q also goes to zero (mod 2n) at Tc-ic- At first 
sight, the system appears to simply evolve from a homogeneous (IC) phase into a 
homogeneous (C) phase as T — > T c ~ic from below. If this picture were correct, 
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here is what histograms would yield: at low T, P(A) would display two maxima at 
±A° (corresponding to the two possible handedness of the spiral in the (IC) state). 
As T — > Tc-ic, the two peaks would merge into a single peak, and, for T > Tc-ic, 
-P(A) would be a gaussian, centered at A^ = 0. By EqsJ^ and [8], we would expect 
7 ra > for T > Tc-ic- 

By contrast, here is what our simulation yields: at low T we do get the two 
maxima at ±A° and as T — > Tc-ic they move closer to each other. However, they 
do not merge: the peaks at ±A° remain sharp and in addition a third peak develops 
at A x = 0, such that A-histograms show a three-peak structure for T > Tc-ic- 
There is a central peak at A^ = and two side peaks centered at T dependent, 
finite values ±Ao- For sizes 48 2 and for simulations using large enough MCS/spin 
the relative weight of the lateral peaks compared to the central peak is roughly 
one. Furthermore, this structure of the A-histogram is observed in a wide range 
of temperatures above Tq-ic- For instance we show the histogram for T > Tc-w 
(Fig (f|)). The structure of -P(A) could have two origins : it could be associated 
with a first order transition, and the fact that the multi-peak structure survives for 
T > Tc-w could be linked to hysteresis effects, or it could be due to the occurence 
of a non homogeneous thermodynamic phase. 

The first order scenario is at variance with the observed temperature dependence 
of j xx in two respects: 

(i) for T — > Tc-w from below, both •y xx and Xo- 1 S° continuously to zero, as 
indicated by Eq.|l3|. (ii) For T > T C -ic up to the paramagnetic boundary, we find 
that j xx = (see Fig (||)); if we tried to explain this property in the framework of 
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a first order transition, this would mean that the system is in a spinodal state over 
a wide range of temperature, which is rather unlikely. 

Instead, we suggest that these data can be consistently interpreted if one consid- 
ers the possibility of a thermodynamically inhomogeneous phase for T > Tc-ic- We 
mentioned in the introduction that systems with competing interactions may lead 
to inhomogeneous groundstates consisting of ordered domains separated by domain 
walls§l~Il. Our simulations reveal that the commensurate phase of the row model 
may well be such an example of stripe phases. 

From the shape of P(Q X ) we see that a measure of (the equilibrium value of) 
Q x at any point of the lattice gives with probability |, +Qo with probability | 
and — Qo with probability j (+Qo and —Qo are the secondary maxima of P(Q X ), 
see Fig (^)) The connection between the equilibrium value of Q x and the plaquette 
chirality (Eq.|II]) implies that the chirality of any site of a given sublattice will be 
positive, negative and zero with probability |, | and | respectively: indeed, if we 
had a homogeneous phase characterized by Q x = +Qo over the entire system, the 
chiralites of the plaquettes of a given sublattice -denoted by A- would all have the 
same sign, say positive; similarly, if we had Q x = —Qo over the entire system, the 
chirality of A would be negative for all the plaquettes; lastly, if Q x = over the 
entire system, the chirality of A would be zero for all the plaquettes. Since the 
values of Q x are distributed according to P(Q X ), we deduce the above mentioned 
distribution for the chiralities of any site of A. 

Fig (|5]) precisely confirms this analysis. It is a plot (as a function of T) of the 
staggered chirality versus T (Eq.|TUD and of the absolute value of the chiralities 
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(where we replace J2(ki)£P°ki by Abs(J2(ki)eP a ki) in Eq.(g)), for 77 = 0.575. These 
two quantities give access to the number of plaquettes with positive, negative and 
zero chirality one each sublattice (see Ref . |I0|) . We see that, for T = 0.4 J, well above 
the (C)-(IC) transition temperature, in what should be the commensurate phase i.e 
a state with zero chirality, 25% of the plaquettes have a positive chirality, 25% of 
the plaquettes have a negative chirality, and 50% of the plaquettes have no chirality. 
With these weights, averaging Q x over the system yields Q x = 0. 

The shape of P(Q X ) also signals the breakdown of the fluctuation-dissipation 
theorem. The correct procedure required to extract the value of j xx is to average 
6 fffi over the distribution -P(A). If the dominant contribution to -P(A) comes from 
a single value, A = A , this yields Eq.[], which gives the most probable value of r ) xx . 
If -P(A) has a multi-peak structure, as is the case here, Eq.|] is not valid: choosing 
for A the value of A corresponding to Q x = +Qo, or to Q x = —Qo, or to Q x = 
which is the mean value of -P(A), would give different values for ^ xx but such that 
■y xx > . By contrast, the average of S over -P(A) leads to •y xx = (see Fig 
(|D). So averages and most probable values do not coincide. 

The picture that emerges from the previous results is that of an inhomogeneous 
structure for T > Tq-ic'- domains of the spiral phase with pitch +Qo coexist with 
domains of the spiral phase with pitch — Q , and the two types are separated by 
domain walls of the collinear phase. It is known that the transition from a homo- 
geneous phase (the (IC) state) to a domain structure can be continuousii, which is 
consistent with our results. The spatial configuration of the domains is visualised 
in Fig (HI), which is a snapshot of the chiralities for r\ = 0.575 and T = OA J. The 
morphology of the state is that of a striped phase. Note that the normal to the 
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direction of the stripes correlates with x (the direction of the 7y-bonds). We dub this 
structure a smectic-like phase: it is solid-like along y {^ yy > 0) but has no rigidity 
along x ( r ) xx = 0); its effective free-energy in the hydrodynamic limit is similar to 
that of a smectic system (Ref(|34|)). 

B. Study of the (C)-(IC) line at fixed T 

To map out the domain of stability of the striped phase in the (rj, T) plane, we 
keep T fixed and we vary 77. Fig (^]) shows j xx versus 77 for T = 0.2J and T = 0.4J; 
in the region delimited by lines AC (77 = 0.5) and AL we get ^ xx = and one expects 
a striped phase there. In other words AL separates an incommensurate phase from 
an inhomogeneous, non-collinear state. 

We have also sought for an analytical evidence of the inhomogeneous state in 
region ALC of the phase diagram Fig ([!]). The next section presents results using 
NSCHA, a variational technique appropriate for frustrated systems: if one seeks 
a uniform collinear solution in region ALC, one finds that ^i xx < 0; this behavior 
stems from the fact that the system is thermodynamically unstable with respect to 
the formation of domains having either Q x = +Qo or Q x = —Qo, the two types 
connecting via domain walls of the collinear (Q x = 0) phase. The breakdown of 
linear response and the properties of ^ xx are hallmarks of the physics of dipolar 
magnets and of spin glasses&~0. 



IV. NSCHA FOR THE COMMENSURATE AND 
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INCOMMENSURATE REGIMES 



In a previous paper we introduced the new self-consistent harmonic approxima- 
tion (NSCHA)§, a variational technique appropriate for frustrated systems. The 
main feature of this approach is that it preserves the coupling between the chiral 
ground states of the system, and that it takes long wavelength chiral fluctuations into 
account. Chiral and phase (spin angle) variables remain coupled at all temperature 
T. We now apply this method to the row model. 



A. The NSCHA variational method. 

The Hamiltonian for XY spins characterized by spin angles reads 

H=-J2 J ij cos ( i- j) ( 14 ) 

(ij) 

where the are nearest neighbor interactions. For frustrated systems the sign of 
the product of the Jij over the links of a plaquette P is negative and this may lead to 
non-collinear configurations in thermal equilibrium. The variational method seeks 
to approximate H (Eq.[l4]) by an harmonic Hamiltonian Hq . We rewrite the $i in 
Eq.0 as 

Bi = 0° + (Pi (15) 

with 0? = (6i) Ho and 

Z (ij) 

Hamiltonian Eq.[L4] is then mapped onto the NSCHA effective hamitonian HnschJ^, 
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H NSC HA = ~ E J ij COS ( 6 i ~ d j) C0S (fi ~ <Pj) ~ ^ EE J V J M ( 17 ) 

x sin (fl? - of) sin - 6»f) sin fa - ^) sin (yj fe - yjj) 

We then average Eq.[l7] over H Q (Eq.[Tj]) and minimize with respect to the varia- 
tional parameters^ 6® and J^- to obtain the NSCHA variational equations. 

In this ensemble we can compute the spinwave stiffness matrix. Its eigenvalues 
are ^Wscha an d 1 V nscha '■ 

Inscha = ^E JiJ cos (fi? - 0°) (%<4)V^/ 2 (18) 
EE JiiJ«(^i■^4)(wfc^^^)e-^^■^ +I ' ^I - Wi - Wk)/2 

x cos (e° - ef cos (el - ef} + sin (e° - ef sin (e° k - e°)~ 

where u x is the unit vector in the horizontal direction, is the vector connecting 
nearest neighbor sites i and j and = ((^ — ipj) 2 ) Ho For j^scha we replace u x 
by Mj, the unit vector in the vertical direction. 

B. NSCHA for the row model 

Applying NSCHA to the row model gives two types of solutions 
a) Commensurate solutions: 
They are characterized by 

0° - 0? = (mod27r) (19) 



with 



2-7T 

Q° x = 0; Q° y = ^ (mod2vr) (20) 
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and by nearest neighbor couplings J^-. There are only two independent interactions 
namely Jij = fjJ for i and j along the horizontal direction, and Jjj = J otherwise. 
These satisfy the following equations 

J= Je-^^- 1 ^)" 172 ] (21) 
f,J=-r ] Je-^ t ^ {1+2 * ) ~ 1/2 } (22) 



Eqs.|2T| and ^ can be self-consistently satisfied without restriction for 77 < 1/2. 



However, if 77 > 1/2 equations Eqs.[H| and [22] have no solution when T < - ln(2r/); 
this was to be expected, since the stable state of the system is a spiral structure at 
low T, for 77 > 1/2. 

b) Incommensurate solutions: 
They correspond to 

0° - 0° = Q .^- ( mo d27r) (23) 

with 

Q°=Q (T); Q°y = ~^ (mod27r) (24) 

The variational equations can only be solved numerically. Just as for the FFTXY 
model, the are no longer short range interactions (for large R, ~ 1/ |r, — r}| 6 
see RefflTDD) and the sign of varies with the relative orientation of i and j. 

Knowledge of the allows us0 to compute the free-energy, Qo(T) , 'jffscHA 5 
1 V nscha (Eq-Pf) and the staggered chirality o~nscha as a function of T for all rj. 

For all rj < 0.5 the lowest free-energy is obtained for the commensurate solution 
and up to the (C)-(P) boundary (line CD in Fig (fj)) ^scha > 0- 
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For 0.5 < rj < rj L , the (IC) solution has the lowest free-energy at low T (T < 
Tc-ic)- As seen on Figs (§) and (^|), NSCHA and MC results agree closely except in 
the vicinity of Tq-ici where defects are expected to play an important role (see our 



previous paper Ref.|10|). In that regime "fffscHA > 0- For T > Tc-ic the variational 
equations favor a commensurate configuration, but we find that jffscHA < ; the 
solution is thermodynamically unstable. By this we mean that NSCHA yields a 
(C) solution in region ALC of the phase diagram, but that fluctuations around the 
solution (given by J^scha) generate an instability. 



To summarize our results, 

1. in the rj, T plane, the transition between the spiral phase and the (C) phase is 
only seen at point A (that is, at zero temperature). Consequently, the Lifshitz 
point is at T = for the 2D XY model. 

2. the existence of the striped phase suggests that chiral variables and phase 
variables remain strongly coupled at all T. This may explain why, despite the 
fact that j xx = in the striped phase, vortices do not unbind (leading to a 
reentrant paramagnetic phase). The relevance of this coupling had already 
been emphasized in our study of the fully frustrated case (rj — 1). 

3. the existence of the inhomogeneous state affects scaling analyses near the (IC)- 
stripe phase boundary (line AL): Fig (§) shows the Binder order parameter 
Eq.(|l2|) as a function of T for 77 = 0.575. We do not observe a clear intersection 



at the critical temperature. A similar feature had been pointed out by Olsson 
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in his study of fully frustrated XY spins on a 2D square lattice (Ref.[l2|). 

4. because the striped phase is spatially inhomogeneous, it is not easy to define 
appropriate boundary conditions for the MC simulation. Uniform twists will 
produce frustration. 

The present work has revealed the existence of a smectic-like phase. This raises 
the question of the nature of the transition between the striped phase and the (P) 
phase (line LC) and also between the striped phase and the commensurate phase 
(line AC); for instance, if the transition line LC is not KT-like, one also needs to 
understand the nature of the critical regime along CD: for rj << 0.5 one recovers a 
KT-transition so there has to be some cross-over. Work is in progress to clarify that 
issue. 

V. CONCLUSION 

Using detailed Monte Carlo simulations we have studied the commensurate- 
incommensurate transition of the two dimensional XY model on a triangular lattice. 
Our study shows that this transition only occurs at T = 0. At finite temperature, 
the incommensurate structure evolves into a striped phase made up of domains of 
left- and right-handed spirals separated by walls. The domain walls consist of the 
collinear structure. This state resembles the smectic-A phase of liquid crystals. The 
nature of the phase transitions between the striped phase and the ordered phases or 
between the striped phase and the paramagnetic phase is an open problem. Analyt- 
ical calculations using NSCHA ( a variational approach well suited for non collinear 
structures ) support the MC results. The connection between frustrated XY models 
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and the vortex state of layered type II superconductors suggest to view the smectic 
phase as a vortex liquid state. This regime would appear to be an intermediate 
phase between the superconducting and the metallic states, critical in one subspace 
and quasi-ordered in the other. 
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FIG. 2. Monte Carlo stiffnesses in the x and y (inset) directions versus T for the 
row model when ij = 0.575. Triangles represent MC data, solid lines are the NSCHA 
predictions. 
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FIG. 3. Qo(T) versus T for r\ = 0.55. Filled circles represent MC data and the solid 
line is the NSCHA prediction. 
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FIG. 4. P(Q X ) versus Q x for 77 = 0.55 and T = 0.19J. 
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FIG. 5. MC determination of the plaquette chirality J2{ki)^p a ki (filled diamonds) 
and of the absolute value of the plaquette chirality Abs(J2{M)&p a ki (open circles) versus 
T for i] = 0.575. 
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FIG. 6. Snapshot of chiralities on each plaquette of a 36 2 triangular lattice. r\ = 0.575 
and T = 0.4J. Filled circles represent plaquettes with the correct sign, i.e in the same 
chiral state as at T = 0. Open circles correspond to plaquettes with the wrong sign, that 
is such that the chirality has changed compared to T = 0. Plaquettes with zero chirality 
(no symbol) are obtained in-between the two. One clearly sees a stripe structure of filled 
circles and open circles separated by domain walls of zero chirality. 
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FIG. 7. MC data for ~f xx , versus 77. The lattice size is 48 2 . ^ xx is obtained from 
the histogram in A modulo The region where 7 :r:r = corresponds to the domain of 
stability of the stripe phase. T is fixed: T = 0.4 J (inset T = 0.2 J). 
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